Skip to content

chore: move all HTTP downloads from Python to Rust; drop httpx (closes #63) - #64

Merged
zackees merged 5 commits into
mainfrom
chore/issue-63-rust-downloaders
May 13, 2026
Merged

zackees merged 5 commits into
mainfrom
chore/issue-63-rust-downloaders

Conversation

@zackees

@zackees zackees commented May 13, 2026

Copy link
Copy Markdown
Member

Summary

Continues from PR #62. Moves the four remaining httpx callsites in Python into Rust install primitives, then drops httpx from pyproject.toml. Python runtime deps: 2 → 1 (zcmds_win32, Windows-only).

Closes #63.

Commits

  1. chore(open_browser): drop redundant wait_for_server HTTP probe — the Rust server prints its listening URL on stdout; the follow-up httpx GET added nothing.
  2. chore(rust): own emscripten install in the Rust CLI — install::ensure_emscripten_installed() does manifest fetch + multipart SHA-256 + .tar.zst extract + .emscripten config. Wired into compile_and_serve; exports FASTLED_EMSCRIPTEN_DIR. Uses the ctcb-manifest crate. Drops clang-tool-chain and pyzstd.
  3. chore(rust): own esbuild install in the Rust CLI; drop fasteners — install::ensure_esbuild_installed() downloads the npm tarball and plucks the bin via the new archive::extract_member_from_tgz. Exports FASTLED_ESBUILD_PATH.
  4. chore(rust): own FastLED repo download in the Rust CLI — install::ensure_fastled_repo(ref) handles latest-release lookup, commit/tag/branch resolution, zip download, and extraction to ~/.fastled/cache/fastled-{ref}/. Two integration points:
    • --init pre-step (sets FASTLED_LOCAL_REPO_DIR before Python takes over).
    • Hidden --internal-ensure-fastled-repo [REF] flag for direct Python-API callers (Api.project_init / Api.get_examples) — prints the local path and exits.
      Python's project_init.py was rewritten around this; EmscriptenToolchain._download_fastled also routes through it.
  5. chore(rust): own Chrome extension CRX download; drop httpx — install::ensure_chrome_extension(id, name) downloads the CRX with the picky User-Agent / Referer headers, scans for the PK magic, strips the CRX header, extracts the embedded zip. Hidden --internal-ensure-chrome-extension flag for the Python playwright shim.

Dep change

Layer Before this PR chain After
Python runtime (pyproject.toml) 4 deps (httpx, fasteners, clang-tool-chain, zcmds_win32) 1 dep (zcmds_win32; sys_platform == 'win32')
Implicit pyzstd (via emscripten_archive.py) –
Rust workspace – ctcb-manifest = "0.4.1", flate2, dirs

grep -r 'import httpx' src/ returns nothing.

Test plan

  • bash lint — cargo check + clippy + rustfmt + ruff + black + isort + pyright all green
  • bash test — 119 passed, 1 skipped, 0 failures (was 118; Api.project_init test now exercises the Rust path end-to-end in 21s)
  • Cold-cache emscripten install — verify on a machine without ~/.fastled/toolchains/emscripten/...
  • Cold-cache esbuild install — same for ~/.fastled/toolchains/esbuild/...
  • Cold-cache --init wasm — empty ~/.fastled/cache/, expect Rust to fetch FastLED + Python to scaffold
  • Cold-cache C++ DevTools extension — empty ~/.fastled/chrome-extensions/, expect Rust to download + extract
  • CI green

Closes #63
Refs #59

🤖 Generated with Claude Code

zackees added 5 commits May 12, 2026 19:24
The Rust CLI prints its listening URL to stdout once the socket is bound.
spawn_http_server() already parses that URL via regex and uses it. The
follow-up HTTP probe through httpx duplicated that gate without adding
information; the fallback path keeps a 1s sleep for the case where the
URL line wasn't seen in time. Removes one httpx callsite.
The Python compile path used to download and extract the emscripten
toolchain itself (httpx + pyzstd, via toolchain/emscripten_archive.py).
That now lives in Rust:

- crates/fastled-cli/src/install.rs (new) - ensure_emscripten_installed()
  fetches the platform manifest, downloads multipart parts with SHA-256
  verification, extracts the .tar.zst, and writes the .emscripten config
  into ~/.fastled/toolchains/emscripten/{platform}/{arch}/{version}/. Uses
  ctcb-manifest (workspace dep) for the manifest schema and the existing
  archive.rs primitives for download/sha256/extract/config.
- crates/fastled-cli/src/main.rs - compile_and_serve() now calls
  ensure_emscripten_installed() once, before invoking Python, and exports
  FASTLED_EMSCRIPTEN_DIR for the child process to consume.
- crates/fastled-cli/src/archive.rs - dropped #![allow(dead_code)] since
  the module is now wired into the compile flow.

Python:
- src/fastled/toolchain/emscripten.py - _get_clang_tool_chain_emscripten_dir()
  reads FASTLED_EMSCRIPTEN_DIR first; ensure_clang_tool_chain_emscripten()
  is now a thin lookup (the install is upstream). The clang_tool_chain
  PyPI fallback was removed.
- src/fastled/toolchain/emscripten_archive.py - deleted (Rust owns it).
- pyproject.toml - dropped clang-tool-chain runtime dep.

Net: Python runtime deps now httpx, fasteners, zcmds_win32 (Win-only).
Lint clean, 118 unit tests pass.
…#59)

The Python frontend bundling step used to download esbuild from npmjs
itself (httpx + tarfile + fasteners InterProcessLock). That now lives
in Rust:

- crates/fastled-cli/src/install.rs - ensure_esbuild_installed()
  downloads the npm registry .tgz, extracts the single bin into
  ~/.fastled/toolchains/esbuild/{platform}/{arch}/{version}/, returns
  the path.
- crates/fastled-cli/src/archive.rs - new extract_member_from_tgz()
  helper plucks a single file out of a gzipped tarball (needs flate2,
  added to workspace).
- crates/fastled-cli/src/main.rs - compile_and_serve() now also calls
  ensure_esbuild_installed() and exports FASTLED_ESBUILD_PATH for the
  Python subprocess.

Python:
- src/fastled/frontend_esbuild.py - install_esbuild() removed; the
  module now resolves esbuild via FASTLED_ESBUILD_PATH (set by Rust) or
  shutil.which fallback. Drops httpx, tarfile, fasteners imports from
  this file.
- src/fastled/toolchain/emscripten_archive.py was the only other
  fasteners consumer (already gone in the previous commit), so fasteners
  drops from pyproject.toml entirely.

Net: runtime Python deps now httpx, zcmds_win32 (Win-only).
Lint clean, 118 unit tests pass.
Python's project_init.py used httpx to query the GitHub API for the latest
release tag and to download the FastLED repo zip. That logic now lives in
Rust:

- crates/fastled-cli/src/install.rs - ensure_fastled_repo(ref: Option<&str>)
  - latest-release lookup via api.github.com (User-Agent header set)
  - resolve_fastled_ref: handles commit SHAs, tags, branches; HEAD-checks
    a tag URL before falling back to a branch URL (mirrors Python)
  - download via archive::download, extract via archive::extract_zip
  - cache at ~/.fastled/cache/fastled-{ref}/ (cheap library.json check on
    subsequent calls)
- crates/fastled-cli/src/main.rs - two integration points:
  - --init pre-step: when --init is set, Rust calls ensure_fastled_repo
    (deriving the ref from --latest / --commit / --branch) and exports
    FASTLED_LOCAL_REPO_DIR before delegating to Python.
  - --internal-ensure-fastled-repo [REF] hidden flag: prints the local
    path to stdout for the Python-driven Api.project_init / Api.get_examples
    paths to consume via subprocess.

Python:
- src/fastled/project_init.py rewritten to use either FASTLED_LOCAL_REPO_DIR
  (preferred) or the Rust binary's --internal-ensure-fastled-repo flag.
  Removes the httpx, tempfile-based extract, and DownloadResult helpers
  that used to live here. _find_rust_fastled_cli() walks the workspace
  target/ tree so it returns the Rust binary, not the Python entry-point
  shim that shadows it in .venv/Scripts/.

Lint clean. test_api.py::test_project_init now exercises the Rust path
end-to-end and passes (21s).
Last remaining Python httpx callsites moved into the Rust CLI:

- crates/fastled-cli/src/install.rs - ensure_chrome_extension(id, name):
  GET the Chrome Web Store CRX with the proper User-Agent / Referer the
  endpoint expects, scan the body for the PK\x03\x04 (or PK\x05\x06)
  ZIP magic, strip the CRX header, extract the embedded zip into
  ~/.fastled/chrome-extensions/{name}/. Cached via manifest.json check.
- crates/fastled-cli/src/main.rs - new hidden plumbing flag
  --internal-ensure-chrome-extension EXTENSION_ID NAME that prints the
  install path to stdout (mirrors the FastLED-repo flag).
- src/fastled/playwright/chrome_extension_downloader.py - thin shim that
  subprocesses the Rust binary. ChromeExtensionDownloader class deleted
  (no external callers).
- src/fastled/toolchain/emscripten.py - EmscriptenToolchain._download_fastled
  used httpx to grab FastLED master for native compile when no local repo
  was specified. Now delegates to project_init._ensure_repo_via_rust (the
  Rust binary handles the download) and copies the cached repo into the
  short-name target_dir/repo layout the build pipeline expects.

pyproject.toml: drop httpx. Net runtime deps now: zcmds_win32 (Win-only).

Lint clean, 119 unit tests pass, no `import httpx` left in src/.
@zackees
zackees merged commit 0115e09 into main May 13, 2026
18 of 25 checks passed
zackees added a commit that referenced this pull request May 13, 2026
Sweeps the non-Python items issue #59 flagged that are still on main
after PRs #61, #62, #64, #65, #66, #68 cleaned up the Python side:

* `entrypoint.sh` — Docker container entrypoint. Sourced
  `/emsdk/emsdk_env.sh` (container path) and called
  `fastled_wasm_compiler.cli_update_from_master` (a module from a
  separate package that only exists inside the container).
* `compiler/run.py` — Docker compile-server entry point. Imports
  `fastled_wasm_compiler`, a separate package not in this repo.
* `compiler/debug.sh` — instructed user to run from `/js` (container
  working directory). Empty `compiler/` directory removed.
* `.dockerignore` — only contained cache exclusions
  (`**/.mypy_cache/`, `**/.ruff_cache/`) which belong in `.gitignore`
  if anywhere; no Docker build context this could serve.
* `chrome_vscode_bridge_design_task.md` — one-off design draft that
  opens with "Here's a refined ... file for your repo. Feel free to
  adjust...". Zero references anywhere else.
* `upload_package.sh` — manual PyPI upload helper, superseded by the
  automated `publish_release.yml` workflow. Zero inbound references.
* Empty root-level `fastled/` directory left over from a prior layout.

`lint` script updated to stop passing the (now-deleted) `compiler/` path
to ruff / black / isort / pyright / the keyboard-interrupt checker.

`bash lint` clean. `bash test` — 119 pass.

Verified by grep across `*.py`, `*.rs`, `*.toml`, `*.yml`, `*.sh`,
`*.md`, `Cargo.toml` (excluding `.venv/` and `.claude/worktrees/`):
none of the deleted files are referenced from a live code path or
workflow.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port project_init + chrome extension downloaders to Rust; drop httpx

1 participant